Method and control system for verifying and updating camera calibration for robot control

ABSTRACT

A robot control system and a method for camera calibration verification is presented. The robot control system is configured to perform a first camera calibration, and to control a robot arm to move a verification symbol to a reference location. The robot control system further receives, from a camera, a reference image of the verification symbol, and determines a reference image coordinate for the verification symbol. The robot control system further controls the robot arm to move the verification symbol to the reference location again during an idle period, receives an additional image of the verification symbol, and determines a verification image coordinate. The robot control system determines a deviation parameter value based the reference image coordinate and the verification image coordinate, and whether the deviation parameter value exceeds a defined threshold, and performs a second camera calibration if the threshold is exceeded.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No.16/369,630, entitled “METHOD AND CONTROL SYSTEM FOR VERIFYING ANDUPDATING CAMERA CALIBRATION FOR ROBOT CONTROL” and filed on Mar. 29,2019, which is expressly incorporated by reference herein in itsentirety.

FIELD OF THE INVENTION

The present invention is directed to a method and control system forverifying and updating camera calibration for robot control.

BACKGROUND

As automation becomes more common, robots are being used in moreenvironments, such as in warehousing and manufacturing environments. Forinstance, robots may be used to load items onto or off of a pallet in awarehouse, or to pick up objects from a conveyor belt in a factory. Themovement of the robot may be fixed, or may be based on an input, such asan image taken by a camera in the warehouse or factory. In the lattersituation, calibration may be performed so as to determine a property ofthe camera, and to determine a relationship between the camera and anenvironment in which the robot is located. The calibration may bereferred to as camera calibration, and may generate camera calibrationinformation that is used to control the robot based on images capturedby the camera. In some implementations, the camera calibration mayinvolve manual operation by a person, who may manually control movementof the robot, or manually control the camera to capture an image of therobot.

SUMMARY

One aspect of the embodiments herein relates to performing cameracalibration verification for robot control. The camera calibrationverification may be performed by a robot control system that comprises acommunication interface and a control circuit. The communicationinterface may be configured to communicate with a robot having a baseand a robot arm with a verification symbol disposed thereon, and tocommunicate with a camera having a camera field of view. The controlcircuit of the robot control system may be configured to perform thecamera calibration verification by: a) performing a first cameracalibration to determine camera calibration information, b) outputting afirst movement command to the communication interface, wherein thecommunication interface is configured to communicate the first movementcommand to the robot to cause the robot arm to move the verificationsymbol, during or after the first camera calibration, to a locationwithin the camera field of view, the location being a reference locationof one or more reference locations for verification of the first cameracalibration, c) receiving an image of the verification symbol via thecommunication interface from the camera, which is configured to capturethe image of the verification symbol at the reference location, theimage being a reference image for the verification, d) determining areference image coordinate for the verification symbol, the referenceimage coordinate being a coordinate at which the verification symbolappears in the reference image, and e) outputting a second movementcommand that is based on the camera calibration information to thecommunication interface, wherein the communication interface isconfigured to communicate the second movement command to the robot tocause movement of the robot arm to perform a robot operation.

In an embodiment, the control circuit is configured to perform thecamera calibration verification further by: f) detecting an idle periodduring the robot operation, g) outputting a third movement command tothe communication interface, wherein the communication interface isconfigured to communicate the third movement command to the robot tocause the robot arm to move the verification symbol to at least thereference location during the idle period, h) receiving via thecommunication interface an additional image of the verification symbolfrom the camera, which is configured to capture the additional image ofthe verification symbol at least at the reference location during theidle period, the additional image being a verification image for theverification, i) determining a verification image coordinate used forthe verification, the verification image coordinate being a coordinateat which the verification symbol appears in the verification image, j)determining a deviation parameter value based on an amount of deviationbetween the reference image coordinate and the verification imagecoordinate, the reference image coordinate and the verification imagecoordinate both associated with the reference location, wherein thedeviation parameter value is indicative of a change since the firstcamera calibration in the camera that the communication interface isconfigured to communicate with, or a change since the first cameracalibration in a relationship between the camera and the robot that thecommunication interface is configured to communicate with, k)determining whether the deviation parameter value exceeds a definedthreshold, and l) performing, in response to a determination that thedeviation parameter value exceeds the defined threshold, a second cameracalibration to determine updated camera calibration information.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features, objects and advantages of theinvention will be apparent from the following description of embodimentshereof as illustrated in the accompanying drawings. The accompanyingdrawings, which are incorporated herein and form a part of thespecification, further serve to explain the principles of the inventionand to enable a person skilled in the pertinent art to make and use theinvention. The drawings are not to scale.

FIGS. 1A and 1B depict block diagrams of systems in which verificationof camera calibration is performed, according to embodiments herein.

FIG. 1C depicts a block diagram of a robot control system configured toperform verification of camera calibration, according to an embodimentherein.

FIG. 1D depicts a block diagram of a camera for which camera calibrationis performed, according to an embodiment herein.

FIG. 2 depicts a system that illustrates a robot being controlled basedon camera calibration information obtained from camera calibration,according to an embodiment herein.

FIG. 3 depicts a system for performing camera calibration, according toan embodiment herein.

FIGS. 4A and 4B provide a flow diagram that illustrates a method forperforming verification of camera calibration, according to anembodiment herein.

FIGS. 5A and 5B illustrate a system in which a verification symbol isdisposed on a robot, wherein the verification symbol is used to performverification of camera calibration, according to an embodiment herein.

FIG. 5C depicts an example verification symbol, according to anembodiment herein.

FIGS. 6A-6B depict examples of reference locations at which respectiveimages of a verification symbol is captured, according to an embodimentherein.

FIG. 7A depicts an example of determining a reference image coordinate,according to an embodiment herein.

FIG. 7B depicts an example of determining a verification imagecoordinate, according to an embodiment herein.

FIG. 8 illustrates an example time line for verification of cameracalibration, according to an embodiment herein.

FIG. 9 provides a flow diagram that illustrates an example method forperforming verification of camera calibration, according to anembodiment herein.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and isnot intended to limit the invention or the application and uses of theinvention. Furthermore, there is no intention to be bound by anyexpressed or implied theory presented in the preceding technical field,background, brief summary or the following detailed description.

Embodiments described herein relate to verifying and/or updatingcalibration of a camera used to control a robot, such as a robot used ina warehouse, a manufacturing plant, or in some other environment. Thecalibration may be referred to as camera calibration, and may beperformed by, e.g., a robot control system (also referred to as a robotcontroller) to generate camera calibration information that facilitatesan ability of the robot control system to control of the robot based onimages captured (e.g., photographed) by the camera. For instance, therobot may be used to pick up a package in a warehouse, wherein placementof a robot arm or other component of the robot may be based on images ofthe package captured by the camera. In that instance, the cameracalibration information may be used along with the images of the packageto determine, for example, a location and orientation of the packagerelative to the robot arm of the robot. The camera calibration mayinvolve determining respective estimates of intrinsic parameters of thecamera (which may also be referred to as internal parameters), anddetermining an estimate of a relationship between the camera and itsexternal environment. An intrinsic parameter of the camera may have oneor more parameter values such as a matrix, a vector, or a scalar value.Further, examples of an intrinsic parameter include a projection matrixand a distortion parameter. In an instance, the camera calibration mayinvolve determining the camera's position with respect to a fixedposition in an external environment, which may be expressed as atransformation function representing the relationship between the cameraand the fixed position in the external environment. In some cases, thecamera calibration may be performed with the aid of a calibrationpattern, which may have pattern elements disposed at defined locationson the calibration pattern. The camera may capture an image of thepattern elements of the calibration pattern (also referred to as acalibration image), and the camera calibration may be performed based oncomparing an image of the pattern elements with the defined locations ofthe pattern elements. Camera calibration is discussed in more detail inU.S. application Ser. No. 16/295,940, filed on Mar. 7, 2019 and titled“METHOD AND DEVICE FOR PERFORMING AUTOMATIC CAMERA CALIBRATION FOR ROBOTCONTROL”, the entire content of which is incorporated herein byreference.

As stated above, one aspect of the present disclosure relates toverifying that a camera calibration performed at an earlier point intime is still accurate at a later point in time. The camera calibrationperformed at the earlier point in time may generate camera calibrationinformation that reflects a property of the camera at that point intime, such as an intrinsic parameter of the camera or a relationshipbetween the camera and its external environment at that point in time.In some cases, an earlier camera calibration may lose accuracy over timebecause the property of the camera may change over time. In a firstexample, an intrinsic parameter of the camera may change over time. Sucha change may be caused by, e.g., a temperature change that alters ashape of a housing and/or a lens of the camera. In a second example, arelationship between the camera and its external environment may changeover time. For instance, the camera may shift in position or orientationrelative to, e.g., a base of the robot or a location in a warehouse.Such a change may be caused by, e.g., a temperature change that expandsor contracts any component used to mount the camera, by a person orother object bumping into the camera, by a vibration in the camera'sexternal environment (e.g., a warehouse), by a force from the camera'sown weight (i.e., by gravity), or by some other factor. These changesmay render the camera calibration information outdated, and using thiscamera calibration information to position a robot arm or othercomponent of the robot at a later point in time may lead to errors. Inother words, if a property associated with the camera has changed overtime but the camera calibration information is not updated to reflectsuch a change, the robot may operate based on outdated or otherwiseincorrect camera calibration information, thereby causing undesirableerrors in the robot's operation. To address the possibility that changesin one or more properties of the camera may occur, a robot controlsystem may automatically perform a verification that detects when cameracalibration information from a camera calibration is no longersufficiently accurate. Detecting such a condition may provide anindication of a change in a property of the camera. If the verificationdetects that the camera calibration information is no longersufficiently accurate, then the robot control system may perform thecamera calibration again to determine updated camera calibrationinformation, which may reflect a more current property or properties ofthe camera. The updated camera calibration information may be used tocontrol placement of the robot arm or some other aspect of the robot'soperation. Accordingly, an automatic verification of the cameracalibration and/or update of the camera calibration are performed toensure that the robot operates based on correct information about one ormore properties associated with the camera.

One aspect of the embodiments herein relates to verifying a cameracalibration for a camera by comparing a reference image captured by thecamera and a verification image captured by the camera. In someinstances, the reference image may be an image of an object capturedwhen the object is at a particular location at an earlier point in time,and the verification image may be an image of the object captured at alater point in time at the same location. The verification may determinewhether there is too much deviation between the reference image and theverification image, such as whether the deviation exceeds a certainthreshold. In some implementations, the object may be a verificationsymbol. More specifically, a robot arm or other component of the robotmay have a verification symbol that is used to verify the cameracalibration. Both the reference image and the verification image maycapture or otherwise include the verification symbol, and the robotcontrol system may compare the two images by comparing an appearance ofthe verification symbol in the reference image with an appearance of theverification symbol in the verification image. For instance, after therobot control system performs a camera calibration that produces cameracalibration information at a particular point in time, the robot controlsystem may control the robot arm (e.g., via a movement command) to movethe verification symbol to a set of predefined locations within thecamera's field of view (also referred to as a camera field of view ofthe camera), wherein these locations may be used as a set of referencelocations for the verification. The camera may capture respectivereference images of the verification symbol at the set of referencelocations. In some cases, the reference images may be capturedimmediately after camera calibration is performed. The movement of therobot arm, or more specifically a movement command used to move therobot arm, may be based on the camera calibration information from thecamera calibration that was just performed, or may be independent of thecamera calibration information. In some cases, the reference images maybe captured before the robot begins robot operation. After the referenceimages are captured, the robot may be considered ready to begin a robotoperation to perform a task, and the robot control system may, e.g.,control positioning of the robot arm based on images subsequentlycaptured by the camera.

As stated above, the reference images may be compared againstsubsequently captured verification images. In an embodiment, theverification images may be captured during one or more idle periodsdetected by the robot control system. More specifically, when the robotoperation begins, the robot may start performing robot tasks (e.g., byinteracting with packages or other objects). While the robot isperforming the robot operation, the robot control system may detect oneor more idle periods for the robot. In some instances, the idle periodmay be a time period during which the robot is free from performing arobot task during the robot operation. In some cases, the robot controlsystem may schedule the robot operation based on detecting or otherwiseanticipating objects that the robot needs to interact with, and maydetect an idle period based on detecting or otherwise anticipating anabsence of objects with which the robot needs to interact.

During the idle period(s), the robot control system may control therobot arm or other component of the robot (e.g., via a movement command)to move to the reference locations and capture (e.g., via a cameracommand) a respective verification image at each of the referencelocations. If the robot has a verification symbol disposed thereon, therobot control system may more specifically control the robot arm to movethe verification symbol to the reference locations to capture theverification images. Subsequently, the robot control system maydetermine how much the respective verification images deviate fromcorresponding reference images at respective reference locations. Insome cases, the deviation between the verification images and respectivereference images may be expressed as a deviation parameter. If a valueof the deviation parameter (also referred to as a deviation parametervalue) exceeds a defined threshold for the deviation parameter (whichmay also be referred to as defined deviation threshold), the robotcontrol system may perform an additional camera calibration to determineupdated camera calibration information for the camera. When a value ofthe deviation parameter exceeds the defined deviation threshold, thiscondition may indicate that use of old camera calibration informationmay lead to an undesirable amount of error in the robot operation. Thus,in some cases, the robot operation may be paused or stopped while theadditional calibration is performed (the pause may be considered anotheridle period). After the additional calibration is complete, a new set ofreference images may be captured, and the robot operation may continuewith the updated camera calibration information. During a subsequentidle period(s), a new set of verification images may be captured, andthe robot control system may perform verification of the additionalcamera calibration by comparing the new set of reference images with thenew set of verification images.

As stated above, if a value of the deviation parameter exceeds thedefined deviation threshold, the robot control system may perform anadditional camera calibration. If the value of the deviation parameterdoes not exceed the deviation threshold, then the robot operation maycontinue after an idle period without the robot control systemperforming an additional calibration. In this scenario, the camera maycapture a new set of verification images at the respective referencelocations during a subsequent idle period(s). When the new set ofverification images are captured, the robot control system may againperform verification of the camera calibration by determining how muchthe new set of verification images deviate from respective referenceimages at respective reference locations.

As stated above, the robot arm may have a verification symbol, such as aring pattern, disposed thereon, and the verification symbol may becaptured by or otherwise included in the reference images and theverification images. In an embodiment, the robot control system maydetermine a deviation between the reference images and the respectiveverification images based on respective locations where the verificationsymbol appears in the reference images, and based on respectivelocations where the verification symbol appears in the verificationimages. For instance, the robot control system may determine a referenceimage coordinate for each reference location. The reference imagecoordinate for a particular location may be a coordinate at which theverification symbol appears in a reference image that was captured whenthe verification symbol was placed at that reference location. Morespecifically, the reference image coordinate may be associated with aparticular reference location, and may refer to an image coordinate atwhich a verification symbol appears in a reference image, wherein thereference image was captured by a camera when the verification symbolwas placed at that reference location. In the above example, the imagecoordinate may refer to a coordinate in an image, such as a pixelcoordinate. When the robot control system subsequently places theverification symbol at a particular reference location again and obtainsa corresponding verification image, the robot control system maydetermine a verification image coordinate. The verification imagecoordinate may also be associated with the reference location, and mayrefer to an image coordinate (e.g., a pixel coordinate) at which theverification symbol appears in the verification image, wherein theverification image was captured by the camera when the verificationsymbol was placed at the reference location. The robot control systemmay compare the reference image coordinate associated with a particularreference location with a verification image coordinate associated withthe same reference location. This comparison may be done for eachreference location at which a verification image and a reference imagewere captured.

In an instance, a reference image coordinate at which a verificationsymbol appears in a reference image may be a coordinate of a center ofthe verification symbol in the reference image (also referred to as acenter coordinate of the verification symbol in the reference image).Similarly, a verification image coordinate at which the verificationsymbol appears in a verification image may be a coordinate of a centerof the verification symbol in the verification image (also referred toas a center coordinate of the verification symbol in the verificationimage). For each reference location at which the robot arm and/orverification symbol was located when a corresponding verification imagewas captured, the robot control system may determine a deviation betweenthe reference image coordinate associated with the reference locationand the verification image coordinate associated with the same referencelocation. If the robot arm and/or verification symbol had been placed ata plurality of reference locations, the robot control system maydetermine respective amounts of deviation between respective referenceimage coordinates and respective verification image coordinates for theplurality of reference locations. The robot control system may furtherdetermine a value of a deviation parameter based on the respectiveamounts of deviation between the reference image coordinates and therespective verification image coordinates for respective referencelocations.

In an instance, the verification symbol may include multiple shapes thatare concentric with one another, such that the respective centers of themultiple shapes in the verification symbol are at the same orsubstantially the same location. For instance, the verification symbolmay be a ring pattern that includes two or more concentric circles. Insome cases, if the reference image coordinate of a verification symbolis a center coordinate of the verification symbol in a reference image,the robot control system may determine a center coordinate of theverification symbol based on respective center coordinates of themultiple shapes in the reference image, wherein the center coordinate ofa particular shape is a coordinate of a center of that shape. If theverification symbol is a ring pattern, the center coordinate of the ringpattern in a reference image may be determined as an average of a centercoordinate of a first circle forming the ring pattern and a centercoordinate of a second circle forming the ring pattern in the referenceimage. Similarly, a center coordinate of a verification symbol in averification image may be determined based on respective centercoordinates of the multiple shapes forming the verification symbol inthe verification image. In some cases, using multiple shapes to form theverification symbol may improve an accuracy of the verification. Forinstance, using the respective center coordinates of the multiple shapesin an image to determine a center coordinate of the verification symbolmay improve a robustness of the verification against image noise. Morespecifically, if an image of the verification symbol includes imagenoise, the image noise may reduce an accuracy by which the robot controlsystem detects a center coordinate of a particular shape of theverification symbol. However, if the center coordinate of that shape isaveraged with a center coordinate of another shape to determine a centercoordinate of the verification symbol, the averaged center coordinatemay reduce an impact of the image noise. As a result, an accuracy indetermining the center coordinate of the verification symbol may beimproved.

In an instance, the verification symbol may have multiple regions withdifferent respective colors, wherein respective areas of the multipleregions may have a distinct and defined ratio. For example, theverification symbol may have a first region having a first color (e.g.,black) and a second region having a second color (e.g., white), where aratio of an area of the first region to an area of the second region isdefined or otherwise known. The distinct ratio may facilitateidentifying the verification symbol in an image, especially if the imagecaptures or otherwise includes other features, such as dots of acalibration pattern. For example, the robot arm that is moving theverification symbol may also have the calibration pattern disposed onthe robot arm. The robot control system may use the ratio to distinguishthe verification symbol from the dots of the calibration pattern. Morespecifically, because the ratio of the areas of the multiple regions ofthe verification symbol is defined as a distinct ratio, the robotcontrol system may identify the verification symbol in an image based onthe defined ratio. During identification of the verification symbolappearing in an image, the robot control system may distinguish theverification symbol from the calibration pattern or other feature basedon the defined ratio. In some cases, the verification symbol may beidentified in the image as a portion of the image having the multipleregions of different respective colors and having the defined ratiobetween respective areas of the multiple regions. If the robot controlsystem or other system or device determines that a particular portion ofthe image does not have multiple regions with different respectivecolors, or that respective areas of the multiple regions have a ratiodifferent than the defined ratio, the robot control system may determinethat the portion of the image is not the verification symbol.

In an instance, the robot control system may perform the verificationbased on a temperature surrounding the robot. For instance, the robotcontrol system may adjust the defined deviation threshold (i.e., definea new value for the deviation threshold) based on the temperature. Forexample, a temperature may affect various parts in the camera and/or inthe robot, as some materials may be sensitive and/or may expand/contractbased on a temperature. A change in temperature may cause an intrinsicparameter(s) of the camera to change, and/or cause a relationshipbetween the camera and its external environment to change. In anembodiment, the deviation threshold may be set to have a first valuewhen the temperature is outside of a defined range, while the deviationthreshold may be set to have a second value lower than the first valuewhen the temperature is within the defined range. For example, when thetemperature is within a defined normal operating temperature range(e.g., within 10 degrees of the room temperature), then the deviationthreshold may be the first value. When the temperature is outside thenormal operating temperature range, then the deviation threshold mayhave the second value lower than the first value. The second value maybe lower than the first value so as to more easily trigger an additionalcamera calibration when the temperature is outside of the normaloperating range, as the temperature outside the normal operatingtemperature range may be more likely to cause changes to the camera orto its relationship with the external environment, and thus more likelyto cause errors in operating the robot with old camera calibrationinformation.

In an embodiment, the verification of the camera calibration may rely ononly a single reference location. Alternatively, the verification of thecamera calibration may rely on a plurality of reference locations. Thereference locations may be any locations in a camera's field of view, ormay be specific, defined locations. For instance, the referencelocations may be defined as locations on a surface of at least oneimaginary sphere that is concave with respect to the camera. At eachreference location in this scenario, the robot arm may be controlled toposition the verification symbol such that the verification symbol ispositioned tangentially to the surface of the at least one imaginarysphere while facing the camera. This positioning may better allow theverification symbol to be photographed or otherwise captured head-on bythe camera (with the verification symbol directly facing the camera), sothat an image of the verification symbol resembles a top view ratherthan a perspective view of the verification symbol. For instance, if theverification symbol is a ring pattern, positioning the ring pattern tobe tangent to the surface of the imaginary sphere may allow a resultingimage of the ring pattern to still appear circular, rather than appearelliptical. The resulting image may exhibit less perspective distortionor no perspective distortion (relative to a scenario in which the ringpattern appears elliptical in the image). The lack of perspectivedistortion may facilitate an accurate determination of a centercoordinate of the ring pattern. In some cases, the reference locationsmay be divided among multiple imaginary spheres that are all concavewith respect to the camera. The multiple imaginary spheres may share acommon center and may be different in size, such that each imaginarysphere has a spherical surface having a different respective distancefrom the camera. In some cases, the camera may be a common center forall of the imaginary spheres.

FIG. 1A illustrates a block diagram of a robot operation system 100(also referred to as the system 100) for performing automatic cameracalibration and automatic verification of the camera calibration. Therobot operation system 100 includes a robot 150, a robot control system110 (also referred to as a robot controller), and a camera 170. In anembodiment, the system 100 may be located within a warehouse, amanufacturing plant, or other premises. The robot control system 110 maybe configured to perform camera calibration, which is discussed in moredetail below, to determine camera calibration information that is laterused to control the robot 150 to perform a robot operation, such aspicking up packages in the warehouse. The robot control system 110 mayfurther be configured to perform camera calibration verification, whichis also discussed in more detail below, to verify whether the cameracalibration information is still sufficiently accurate. In some cases,the robot control system 110 is configured to perform the cameracalibration and to control the robot 150 to perform robot operationbased on the camera calibration information. In some cases, the robotcontrol system 110 may form a single device (e.g., a single console or asingle computer) that communicates with the robot 150 and the camera170. In some cases, the robot control system 110 may include multipledevices.

In some cases, the robot control system 110 may be dedicated toperforming the camera calibration and/or verification of the cameracalibration, and may communicate the most current camera calibrationinformation to another control system (also referred to as anothercontroller, not shown) that then controls the robot 150 to perform arobot operation based on the most current camera calibrationinformation. The robot 150 may be positioned based on images captured bythe camera 170 and on the camera calibration information. Morespecifically, the robot control system 110 may, in an embodiment, beconfigured to generate movement commands based on the images and basedon the camera calibration information, and to communicate the movementcommands to the robot 150 to control movement of its robot arm. In somecases, the robot control system 110 is configured to performverification of the camera calibration during an idle period in therobot operation. In some cases, the robot control system 110 isconfigured to perform the verification while performing a robotoperation with the robot 150.

In an embodiment, the robot control system 110 may be configured tocommunicate via a wired or wireless communication with the robot 150 andthe camera 170. For instance, the robot control system 110 may beconfigured to communicate with the robot 150 and/or the camera 170 via aRS-232 interface, a universal serial bus (USB) interface, an Ethernetinterface, a Bluetooth® interface, an IEEE 802.11 interface, or anycombination thereof. In an embodiment, the robot control system 110 maybe configured to communicate with the robot 150 and/or the camera 170via a local computer bus, such as a peripheral component interconnect(PCI) bus.

In an embodiment, the robot control system 110 may be separate from therobot 150, and may communicate with the robot via the wireless or wiredconnection discussed above. For instance, the robot control system 110may be a standalone computer that is configured to communicate with therobot 150 and the camera 170 via a wired connection or wirelessconnection. In an embodiment, the robot control system 110 may be anintegral component of the robot 150, and may communicate with othercomponents of the robot 150 via the local computer bus discussed above.In some cases, the robot control system 110 may be a dedicated controlsystem (also referred to as a dedicated controller) that controls onlythe robot 150. In other cases, the robot control system 110 may beconfigured to control multiple robots, including the robot 150. In anembodiment, the robot control system 110, the robot 150, and the camera170 are located at the same premises (e.g., warehouse). In anembodiment, the robot control system 110 may be remote from the robot150 and the camera 170, and may be configured to communicate with therobot 150 and the camera 170 via a network connection (e.g., local areanetwork (LAN) connection).

In an embodiment, the robot control system 110 may be configured toretrieve or otherwise receive images of a calibration pattern 160 and/orof a verification symbol 165 disposed on the robot 150 (e.g., on a robotarm of the robot) from the camera 170. In some instances, the robotcontrol system 110 may be configured to control the camera 170 tocapture such images. For example, the robot control system 110 may beconfigured to generate a camera command that causes the camera 170 tocapture an image of a field of view of the camera 170 (also referred toas a camera field of view), and to communicate the camera command to thecamera 170 via the wired or wireless connection. The same command maycause the camera 170 to also communicate the image to the robot controlsystem 110, or more generally to a storage device accessible by therobot control system 110. Alternatively, the robot control system 110may generate another camera command that causes the camera 170, uponreceiving the camera command, to communicate an image(s) it has capturedto the robot control system 110. In an embodiment, the camera 170 mayautomatically capture an image in its camera field of view, eitherperiodically or in response to a defined triggering condition, withoutneeding a camera command from the robot control system 110. In such anembodiment, the camera 170 may also be configured to automatically,without a camera command from the robot control system 110, communicatethe image to the robot control system 110 or, more generally, to astorage device accessible by the robot control system 110.

In an embodiment, the robot control system 110 may be configured tocontrol movement of the robot 150 via movement commands that aregenerated by the robot control system 110 and communicated over thewired or wireless connection to the robot 150. The robot 150 may beconfigured to have one or both of the calibration pattern 160 and theverification symbol 165 on the robot 150. For instance, FIG. 1B depictsa robot operation system 100A in which the verification symbol 165 isdisposed on the robot 150 without the presence of the calibrationpattern 160 of FIG. 1A. In one instance, the verification symbol 165 maybe a part of the robot 150 and may be permanently disposed on the robot150. For example, the verification symbol 165 may be permanently paintedon the robot 150, or may be part of a sticker or board that ispermanently attached to the robot 150. In another instance, theverification symbol 165 may be a separate component that is attachableto and detachable from the robot 150. The verification symbol 165 may bepermanently disposed on the robot 150, or may be a separate componentthat can be attached to and detached from the robot 150.

In an embodiment, the only images used in system 100 to control therobot 150 may be those captured by the camera 170. In anotherembodiment, the system 100 may include multiple cameras, and the robot150 may be controlled by images from the multiple cameras.

FIG. 1B further illustrates an embodiment in which the robot controlsystem 110 is in communication with a user interface device 180. Theuser interface device 180 may be configured to interface with anoperator of the robot 150, such as an employee at a warehouse in whichthe robot 150 is located. The user interface device 180 may include,e.g., a tablet computer or desktop computer that provides a userinterface displaying information relating to operation of the robot 150.As stated above, the robot control system 110 may be configured todetect when a deviation parameter value exceeds a defined deviationthreshold. In an embodiment, the user interface device 180 may providean alarm or other alert to notify the operator of the deviationparameter value exceeding the defined deviation threshold.

FIG. 1C depicts a block diagram of the robot control system 110. Asillustrated in the block diagram, the robot control system 110 includesa control circuit 111, a communication interface 113, and anon-transitory computer-readable medium 115 (e.g., memory). In anembodiment, the control circuit 111 may include one or more processors,a programmable logic circuit (PLC) or a programmable logic array (PLA),a field programmable gate array (FPGA), an application specificintegrated circuit (ASIC), or any other control circuit.

In an embodiment, the communication interface 113 may include one ormore components that are configured to communicate with the camera 170of FIG. 1A or 1B and the robot 150 of FIG. 1A or 1B. For instance, thecommunication interface 113 may include a communication circuitconfigured to perform communication over a wired or wireless protocol.As an example, the communication circuit may include a RS-232 portcontroller, a USB controller, an Ethernet controller, a Bluetooth®controller, a PCI bus controller, any other communication circuit, or acombination thereof. In an embodiment, the control circuit 111 may beconfigured to generate a movement command (e.g., a motor movementcommand) and to output the movement command to the communicationinterface 113. In this embodiment, the communication interface 113 maybe configured to communicate the movement command to the robot 150 tocontrol movement of a robot arm or other component of the robot 150. Inan embodiment, the control circuit 111 may be configured to generate acamera command and to output the camera command (e.g., capture imagecommand) to the communication interface 113. In this embodiment, thecommunication interface 113 may be configured to communicate the cameracommand to the camera 170 to control the camera 170 to photograph orotherwise capture an image of an object in the camera's field of view.In an embodiment, the communication interface 113 may be configured toreceive an image or other data from the camera 170, and the controlcircuit 111 may be configured to receive the image from thecommunication interface 113.

In an embodiment, the non-transitory computer-readable medium 115 mayinclude computer memory. The computer memory may comprise, e.g., dynamicrandom access memory (DRAM), solid state integrated memory, and/or ahard disk drive (HDD). In some cases, the camera calibration may beimplemented through computer-executable instructions (e.g., computercode) stored on the non-transitory computer-readable medium 115. In suchcases, the control circuit 111 may include one or more processorsconfigured to perform the computer-executable instructions to performverification of camera calibration (e.g., the steps illustrated in FIGS.4A, 4B, and 9).

FIG. 1D depicts a block diagram of the camera 170 that includes one ormore lenses 171, an image sensor 173, and a communication interface 175.The communication interface 175 may be configured to communicate withthe robot control system 110 of FIG. 1A, 1B or 1C, and may be similar tothe communication interface 113 of FIG. 1C of the robot control system110. In an embodiment, the one or more lenses 171 may focus light thatis coming from outside the camera 170 onto the image sensor 173. In anembodiment, the image sensor 173 may include an array of pixelsconfigured to represent an image via respective pixel intensity values.The image sensor 173 may include a charge-coupled device (CCD) sensor, acomplementary metal oxide semiconductor (CMOS) sensor, a quanta imagesensor (QIS), or any other image sensor.

As stated above, the camera calibration may be performed in order tofacilitate the control of a robot based on images captured by a camera.For instance, FIG. 2 depicts a robot operation system 200 (also referredto as the system 200) in which the images are used to control a robot250 to perform a robot operation, such as an operation to pick up anobject 292 in a warehouse. More specifically, the system 200 may be anembodiment of system 100 of FIG. 1A, and includes a camera 270, therobot 250, and the robot control system 110. The camera 270 may be anembodiment of the camera 170 of FIG. 1A, 1B, or 1D, and the robot 250may be an embodiment of the robot 150 of FIG. 1A or 1B. The camera 270may be configured to capture an image of the object 292 (e.g., a packagefor shipping) disposed on a conveyor belt 293 in the warehouse, and therobot control system 110 may be configured to control the robot 250 topick up the object 292. When there are one or more objects on theconveyor belt 293, the robot control system 110 may be configured toschedule movement of the robot 250 to pick up the objects. The robotcontrol system 110 may in some cases be configured to detect an idleperiod for the robot operation by detecting when there are no objects onthe conveyor belt 293, or when there are no objects on the conveyor belt293 that are within reach of the robot 250.

In the embodiment of FIG. 2, the robot 250 may have a base 252 and arobot arm that is movable relative to the base 252. More specifically,the robot arm may comprise a plurality of links 254A through 254E, and arobot hand 255 attached to the link 254E. The plurality of links 254Athrough 254E may be rotatable relative to each other, and/or may beprismatic links that are movable linearly with respect to each other.Because FIG. 2 involves the robot 250 that is used to pick up objects,the robot hand 255 may include grippers 255A and 255B used to grab theobject 292. In an embodiment, the robot control system 110 may beconfigured to communicate a movement command to rotate one or more ofthe links 254A through 254E. The movement command may be a low-levelcommand, such as motor movement commands, or a high-level command. Ifthe movement command from the robot control system 110 is a high-levelcommand, the robot 150 may be configured to convert the high-levelcommand to a low-level command.

In an embodiment, the camera calibration information determined from thecamera calibration describes a relationship between the camera 270 andthe robot 250, or more specifically a relationship between the camera270 and a world point 294 that is stationary relative to the base 252 ofthe robot 250. The world point 294 may represent a world or otherenvironment in which the robot 250 is located, and may be any imaginarypoint that is stationary relative to the base 252. In other words, thecamera calibration information may include information describing arelationship between the camera 270 and the world point 294. In anembodiment, this relationship may refer to a location of the camera 270relative to the world point 294, as well as an orientation of the camera270 relative to a reference orientation for the robot 250. The aboverelationship between the camera 270 and the world point 294 may bereferred to as a camera-to-world relationship, and may be used torepresent a relationship between the camera 270 and the robot 250. Insome cases, the camera-to-world relationship may be used to determine arelationship between the camera 270 and the object 292 (also referred toas a camera-to-object relationship), and a relationship between theobject 292 and the world point 294 (also referred to as anobject-to-world relationship). The camera-to-object relationship and theobject-to-world relationship may be used to control the robot 250 topick up the object 292.

In an embodiment, the camera calibration information may describe anintrinsic parameter of the camera 270, where the intrinsic parameter maybe any parameter whose value is independent of a location and anorientation of the camera 270. The intrinsic parameters may characterizea property of the camera 270, such as its focal length, a size of itsimage sensor, or an effect of lens distortion introduced by the camera270.

An example showing a detailed structure of an example of a robot 350 isdepicted in FIG. 3, which depicts a robot operation system 300 thatincludes the robot control system 110 in communication with a camera 370and the robot 350. The camera 370 may be an embodiment of the camera170/270 of FIG. 1A, 1B, 1D, or 2 respectively, and the robot 350 may bean embodiment of the robot 150/250 of FIG. 1A, 1B, or 2 respectively.The camera 370 may be capable of capturing images within a camera fieldof view 330. The robot 350 may include a base 352 and a robot armmovable relative to the base 352. The robot arm includes one or morelinks, such as links 354A through link 354E, and a robot hand 355. In anembodiment, the links 354A-354E may be rotatably attached to each other.For instance, the link 354A may be rotatably attached to the robot base352 via a joint 356A. The remaining links 354B through 354E may berotatably attached to each other via joints 356B through 356E. In anembodiment, the base 352 may be used to mount the robot 350 to, e.g., amounting frame or mounting surface (e.g., floor of a warehouse). In anembodiment, the robot 350 may include a plurality of motors that areconfigured to move the robot arm by rotating the links 354A-354E. Forinstance, one of the motors may be configured to rotate the first link354A with respect to the joint 356A and the base 302, as shown with thedotted arrow in FIG. 3. Similarly, other motors of the plurality ofmotors may be configured to rotate the links 354B through 354E. Theplurality of motors may be controlled by the robot control system 110.FIG. 3 further depicts the robot hand 355 disposed in a fixed manner onthe fifth link 354E. The robot hand 355 may have a calibration pattern320 thereon, such that the robot control system 110 may capture imagesof the calibration pattern 320 via the camera 370 and perform cameracalibration based on the captured images of the calibration pattern 320.For example, the robot control system 110 may move the robot arm suchthat the calibration pattern 320 may be within the camera field of view330 and visible to the camera 370 when the camera 370 is being used tocapture the images of the calibration pattern 320 (also referred to ascalibration images). After the camera calibration is performed, therobot hand 355 may be removed and replaced with another robot hand, suchas a robot hand having a verification symbol disposed thereon, asdiscussed below in more detail.

As stated above, according to an embodiment, the camera calibrationverification may involve comparing a reference image coordinate at whicha verification symbol appears in a reference image to a verificationimage coordinate at which the verification symbol appears in averification image. The comparison may determine a deviation between theverification image coordinate and the reference image coordinate, andthe deviation may be used to determine whether to perform additionalcamera calibration. The verification image may be captured during anidle period of a robot operation. FIGS. 4A and 4B depict flow diagramsthat illustrate a method 400 for verification of camera calibration,according to an embodiment. In an embodiment, the method 400 may beperformed by the control circuit 111 of the robot control system 110. Asstated above, the robot control system 110 may include the communicationinterface 113 of FIG. 1C, which is configured to communicate with therobot 150 of FIG. 1A or 1B, and with the camera 170 of FIG. 1A, 1B, or1D. In an embodiment, the robot may have a base (e.g. the base 252 ofFIG. 2 or the base 352 of FIG. 3) and a robot arm (e.g. the robot arm ofFIG. 2 or FIG. 3) with a verification symbol disposed on thereon, andwherein the robot arm is movable relative to the base.

An example environment in which the method 400 is performed is depictedin FIGS. 5A and 5B, which depicts a robot operation system 500/500A thateach includes the robot control system 110 in communication with acamera 570 and a robot 550. The camera 570 may be an embodiment of thecamera 170/270/370 of FIG. 1, 2, or 3 respectively, and the robot 550may be an embodiment of the robot 150/250/350 of FIG. 1A, 1B, 2, or 3,respectively. The robot 550 may include a base 552 and a robot armmovable relative to the base 552. The robot arm includes one or morelinks, such as links 554A through link 554E. In an embodiment, the links554A-554E may be rotatably attached to each other. For instance, thelink 554A may be rotatably attached to the robot base 552. The remaininglinks 554B through 554E may be rotatably attached to each other via aplurality of joints. In an embodiment, the base 552 may be used to mountthe robot 552 to, e.g., a mounting frame or mounting surface (e.g.,floor of a warehouse). The robot 550 may operate in a similar manner tothe robot 350 of FIG. 3. For instance, the robot 550 may includemultiple motors configured to move the robot arm by rotating the links554A-554E relative to each other. The robot arm may further include arobot hand that is attached to the link 554E. For example, FIG. 5Adepicts a first robot hand 555, a second robot hand 557, and a thirdrobot hand 559, each of which may be attachable to and detachable fromthe fifth link 554E. The robot hand 555/557/559 may include, e.g., agripper or suction device that is configured to pick up objects (e.g.,582A, 582B, 582C) from a conveyor belt 573. When the robot hand555/557/559 is attached to the fifth link 554E, the attachment may be ina mixed manner. The attachment and detachment may be performed manuallyor automatically. In one example, the fifth link 554E may be attached tothe first robot hand 555, as depicted in FIGS. 5A and 5B, and the robotcontrol system 110 may control the robot 550 to cause the fifth link554E to release the first robot hand 555 and attach the fifth link 554Eto the second robot hand 557. In another embodiment, the fifth link 554Emay be permanently attached to a robot hand (e.g., robot hand 559).

In an embodiment, the robot 550 may have a verification symbol 530disposed thereon. In some cases, the verification symbol 530 may bepermanently disposed on the robot 550. In some cases, the verificationsymbol 530 may be disposed on a robot arm of the robot 550, such as oneof the links 554A-554E, or on a robot hand. For instance, FIG. 5Adepicts the verification symbol 530 being disposed on the first robothand 555 and the third robot hand 559, while FIG. 5B depicts theverification symbol 530 being disposed on the link 554E. Theverification symbol 530 may be directly painted on the robot 550, or maybe attached to the robot 550, such as via a sticker or a flat board. Inthe example depicted in FIG. 5A, the second robot hand 557 or the thirdrobot hand 559 may be used to perform camera calibration, because theyeach have a respective calibration pattern 520/527 disposed thereon,while the first robot hand 555 or the third robot hand 559 may be usedto perform verification of the camera calibration, because they eachhave a respective verification symbol 530 disposed thereon.

Returning to FIG. 4A, in an embodiment the method 400 may begin withstep 401, in which the control circuit 111 performs a first cameracalibration to determine camera calibration information associated withthe camera (e.g., camera 170/270/370/570 of FIG. 1, 2, 3, or 5respectively). More specifically, the camera calibration information mayinclude camera calibration values for the camera. In this embodiment,the control circuit 111 may perform the first camera calibration basedon images of a calibration pattern (also referred to as calibrationimages).

For instance, to perform the first camera calibration, the robot 550 ofFIG. 5A may be attached to the second robot hand 557 having thecalibration pattern 520 or the third robot hand 559 having thecalibration pattern 527. FIG. 3 depicts a similar environment in whichthe first camera calibration can be performed. During this step, thefirst robot hand 555 may be detached from the fifth link 554E, in whichthe calibration pattern 320 is used to perform camera calibration. Thefirst camera calibration may be performed before starting a robotoperation. For instance, the robot operation may begin with a robot tasksuch as the first robot hand 555 interacting with a first object 582A ona conveyor belt. During the first camera calibration, the robot 550 maybe equipped with the second robot hand 557. The robot control system 110may control the robot arm of the robot 550, via movement commands, tomove the calibration pattern 520 to various locations within a camerafield of view 510 of the camera 570 and to capture respective images ofthe calibration pattern 520 at such locations. The robot control system110 may perform the first camera calibration to determine the cameracalibration information for the camera 570 based on the captured imagesof the calibration pattern 520. In an example, the camera calibrationinformation may include information describing a relationship betweenthe camera 570 and the robot 550. In an example, the camera calibrationinformation may describe the intrinsic parameters of the camera 570.Camera calibration is discussed in more detail in U.S. application Ser.No. 16/295,940, filed on Mar. 7, 2019 and titled “METHOD AND DEVICE FORPERFORMING AUTOMATIC CAMERA CALIBRATION FOR ROBOT CONTROL,” the entirecontent of which is incorporated herein by reference

Returning to FIG. 4A, the method 400 may further include step 403, inwhich the control circuit 111 controls the robot arm to move theverification symbol (e.g., 530 of FIG. 5), during or after the firstcamera calibration, to a location within a camera field of view (e.g.,510) of the camera (e.g., 570) by outputting a first movement command tothe communication interface 113 of the robot control system 110. Thecommunication interface 113 may be configured to communicate themovement command to the robot to cause the robot arm to move theverification symbol (e.g., 530), during or after the first cameracalibration, to the location within the camera field of view (e.g.,510). The movement command may also cause the robot arm to orient theverification symbol to face the camera (e.g., 570), or more generally tobe visible to the camera. The location may be used as a referencelocation of one or more reference locations for verification of thefirst camera calibration. For instance, when the verification processacquires images of the verification symbol over time, the controlcircuit 111 may control the robot arm to consistently position theverification symbol (e.g., 530) at the same one or more locations, suchthat the one or more locations may be used as one or more referencelocations. Further, as described below with respect to steps 405-459,the verification process may compare later images of the verificationsymbol against a set of earlier images of the verification symbol (e.g.,530), such as images obtained immediately after the first cameracalibration is performed. The later images may be used as verificationimages, while the images against which the later images are compared maybe used as reference images.

In step 405, the control circuit 111 may receive (e.g., retrieve) animage of the verification symbol (e.g., 530) from the camera (e.g.,170/270/370/570) via the communication interface 113, wherein the imageis a reference image for the verification. The image may have beencaptured by the camera while the verification symbol is or was at thereference location. In an embodiment, the communication interface 113may initially receive the reference image from the camera, and thecontrol circuit 111 may then receive the reference image from thecommunication interface 113. In an embodiment, step 405 is performedwithout the control circuit 111 generating a camera command for thecamera. In an embodiment, step 405 may involve the control circuit 111generating a camera command and communicating the camera command to thecamera via the communication interface 113. The camera command maycontrol the camera to capture an image of the verification symbol at thereference location.

FIGS. 5A through 6B illustrate aspects of steps 403 and 405. In theembodiment of FIG. 5A, after the first camera calibration is performedwith, e.g., the second robot hand 557, the second robot hand 557 may bereplaced with the third robot hand 559, which has the verificationsymbol 530 disposed thereon. In this instance, the robot control system110 controls the robot arm of the robot 550 (e.g., via one or moremovement commands) to move the verification symbol 530 to one or morereference locations within the camera field of view 510 of the camera570. The one or more reference locations may include any location withinthe camera field of view 510, or may be a set of one or more specificlocations, such as locations disposed on a surface of an imaginarysphere, as discussed below in more detail. In another instance, in theembodiment of FIG. 5B, during or after the first camera calibration, therobot control system 110 may control the robot arm to move theverification symbol 530 to one or more reference locations within thecamera field of view 510. In this instance, the one or more referencelocations may include any location at which verification symbol 530 wasphotographed (along with the calibration pattern 520) during the firstcamera calibration, or may be a set of one or more specific locations towhich the verification symbol 530 is moved after the first cameracalibration is performed. The robot control system 110 may control themovement of the robot arm of the robot 550 in this step with guidance ofthe camera 570 based on the camera calibration information obtained fromthe first camera calibration, or may do so without such guidance. In anembodiment, the reference locations may be defined locations that arestored in a local or remote storage device and may be retrieved. Theymay be stored in the form of coordinates (e.g., Cartesian coordinates)or as motor commands for rotating the links 554A-554E, or in some othermanner.

In an embodiment, the one or more reference locations to which the robotarm moves the verification symbol (e.g., 530) may include a plurality ofreference locations, wherein each of the plurality of referencelocations is a location disposed on a surface of an imaginary spherethat is concave with respect to the camera. In such an embodiment, thecontrol circuit 111 may be further configured to control the robot armto move the verification symbol to be tangent to the surface of theimaginary sphere at each reference location of the plurality ofreference locations. For instance, as illustrated in FIGS. 6A and 6B,the robot control system 110 may control the robot arm of the robot 550to move the verification symbol 530 to reference locations 610A-610I,and control the camera 570 to capture a respective reference image ateach of the reference locations 610A-610I. The reference locations610A-610I in FIGS. 6A and 6B may be divided among a plurality ofimaginary spheres within the camera field of view 510. The referencelocations 610A and 610B may be disposed on a first spherical surface 621of a first imaginary sphere 620, where the first spherical surface 621is within the camera field of view 510. The reference locations 610C,610D, and 610E may be disposed on a second spherical surface 631 of asecond imaginary sphere 630, where the second spherical surface 631 iswithin the camera field of view 510. The reference locations 610F, 610G,610H, and 610I may be disposed on a third spherical surface 641 of athird imaginary sphere 640, where the third spherical surface 641 iswithin the camera field of view 510. As illustrated in FIGS. 6A and 6B,the first, second, and third spherical surfaces 621, 631, and 641,respectively, are concave with respect to the camera 570. Although theexamples in FIGS. 6A and 6B show three spherical surfaces based on threespheres, a number of different spherical surfaces on which referencelocations may be disposed may be greater than three or less than three.In an embodiment, the camera 570 may be a center of each of theimaginary spheres 620, 630, 640.

In an embodiment, as illustrated in FIGS. 6A and 6B, when theverification symbol 530 is moved to a reference location, the robotcontrol system 110 may control the robot arm of the robot 550 (e.g., viaa movement command) to position the verification symbol 530 to betangent to the spherical surface on which the reference location isdisposed. For example, FIG. 6B illustrates that the verification symbol530 is tangent to the second spherical surface 631 at the referencelocation 610D. More particularly, the verification symbol 530 may bedisposed on a flat plane (e.g., on a sticker), and the flat plane of theverification symbol 530 may be tangent to the second spherical surface631 at the reference location 610D.

In an embodiment, the control circuit 111 is configured to control therobot arm to move the verification symbol (e.g., 530) to directly facethe camera when the verification symbol is moved to a referencelocation. For instance, as illustrated in FIG. 6A, the robot controlsystem 110 may control the robot arm of the robot 550 to move theverification symbol 530 to directly face the camera 570 when theverification symbol 530 is moved to the reference location 610D. In thisexample, the robot control system 110 may control the robot hand 555 tobe rotated such that the verification symbol 530 directly faces thecamera 570. In some cases, the verification symbol may directly face thecamera 570 by being tangent to a spherical surface at the camera fieldof view 510. When the verification symbol 530 directly faces the camera570, the camera 570 may be able to photograph the verification symbol530 head-on, so that there is no perspective effect or a reducedperspective effect in a resulting image of the verification symbol 530.

In an embodiment, the verification symbol (e.g., 530) includes a firstregion having a first color and a second region having a second color,wherein a ratio of an area of the first region to an area of the secondregion is defined and stored on a non-transitory computer-readablemedium (e.g., storage device) of the robot control system 110. In suchan embodiment, the control circuit 111 may be configured to identify theverification symbol in a reference image or a verification image basedon the defined ratio. For instance, as illustrated in FIG. 5C, theverification symbol 530 may include a first region 531 that isring-shaped and has a first color (e.g., black region), and includes asecond region 533 (e.g., white region) enclosed by the first region 531and having a second color. The ratio of an area of the first region 531in black to an area of the second region 533 in white in theverification symbol 530 may be a distinct, defined value. By analyzingthe colors within a captured image, the robot control system 110 may becapable of identifying a portion of the image that corresponds to theverification symbol 530 by determining whether that portion has aring-shaped region that encloses a circular region, and whether a ratiobetween an area of the ring-shaped region and an area of the circularregion matches the defined ratio This may allow the robot control system110 to distinguish the verification symbol 530 from other featurescaptured in an image. For example, as illustrated in FIG. 5A, the robot550 may be set to utilize the third robot hand 559 that has acombination of a calibration pattern 527 and a verification symbol 530.In this example, a reference image may show both the verification symbol530 and the calibration pattern 527. In this example, the calibrationpattern 527 may not have any ring pattern, or may have ring patternswith a different ratio than the defined ratio discussed above. Thecontrol circuit 111 may determine whether a portion of the referenceimage is the verification symbol 530 or the calibration pattern 527 bydetermining whether the portion of the reference image has a first imageregion having the first color and has a second image region having thesecond color, and whether a ratio between an area of the first imageregion and an area of the second image region is equal to the definedratio.

In some cases, the robot control system 110 may determine whether aparticular portion of a captured image has a first region with a firstcolor and a second region with a second color, and whether a ratiobetween an area of the first region and an area of the second region iswithin a defined range. In one example, if the defined ratio is 1.5,then the robot control system 110 may determine that the particularregion corresponds to the verification symbol 530 if the ratio in theparticular region falls within a range between 1.4 and 1.6. The twocolors of the first and second regions are not limited to black andwhite colors and may be any two different colors that can bedistinguishable by the robot control system 110.

In an aspect, the verification symbol (e.g., 530) may include a firstshape and a second shape that are concentric with each other, whereinrespective centers of the first shape and the second shape are atsubstantially the same location. For instance, the verification symbolmay be shaped as a circular ring, which includes a first circle and asecond circle that are concentric with each other. More specifically, asillustrated in FIG. 5C, the verification symbol 530 may include a firstshape 535 (e.g., outer circle) and a second shape 537 (e.g., innercircle). The first shape 535 and the second shape 537 may be concentricto each other, such that a center of the first shape 535 and a center ofthe second shape 537 are at substantially at the same location. Forexample, if a center of the first shape 535 is at a coordinate(u_(symbol) ^(outer), v_(symbol) ^(outer)) and a center of the secondshape 537 is at a coordinate (u_(symbol) ^(inner), v_(symbol) ^(inner)),the coordinate (u_(symbol) ^(outer), v_(symbol) ^(outer)) and thecoordinate (u_(symbol) ^(inner), v_(symbol) ^(inner)) may besubstantially the same.

Returning to FIG. 4A, the method 400 may further include step 407, inwhich the control circuit 111 determines a reference image coordinatefor the verification symbol, the reference image coordinate being acoordinate at which the verification symbol (e.g., 530) appears in thereference image. For instance, as illustrated in FIG. 6A, an image ofthe verification symbol 530 may be captured at the reference location610D, and may be used as the reference image. The verification symbol530 may appear within the reference image at a particular coordinate,which may be referred to as a reference image coordinate.

In an embodiment, as discussed above, the verification symbol (e.g.,530) may include a first shape and a second shape that are concentricwith each other, wherein respective centers of the first shape and thesecond shape are at substantially the same location. In such anembodiment, the control circuit 111 in step 407 may be configured todetermine the reference image coordinate for such a verification symbolby: determining a first coordinate of a center of the first shape in thereference image, determining a second coordinate of a center of thesecond shape in the reference image, and determining the reference imagecoordinate as an average of the first coordinate and the secondcoordinate in the reference image.

For instance, FIG. 7A shows a reference image 710 captured at areference location N (where N is an integer) of the reference locations.The reference image 710 includes a verification portion 730, which is animage portion in the reference image 710 showing the verification symbol530 of FIG. 5A, 5B, or 5C. The robot control system 110 of FIG. 1A or 1Bmay be configured to identify, from the verification portion 730, afirst shape 735 (e.g., outer circle) that is the same as orsubstantially the same as the first shape 535 of the verification symbol530 of FIG. 5C. The robot control system 110 may be configured tofurther identify, from the verification portion 730, a second shape 737(e.g., inner circle) that is the same or substantially the same as thesecond shape 537 of the verification symbol 530 in FIG. 5C.Subsequently, for the reference location N, the robot control system 110may determine a first coordinate (u_(ref _N) ^(outer), v_(ref _N)^(outer)) of a center of the first shape 735 (i.e., a center coordinateof the first shape 735) shown in the reference image 710 and a secondcoordinate (u_(ref _N) ^(inner), v_(ref _N) ^(inner)) of a center of thesecond shape 737 (i.e., a center coordinate of the second shape 737)shown in the reference image 710. To determine the reference imagecoordinate (u_(ref _N), v_(ref _N)) for the reference image 710 as awhole, wherein the reference image 710 corresponds with the verificationsymbol 530 being at reference location N, the robot control system 110may calculate an average of the first coordinate (u_(ref _N) ^(outer),v_(ref _N) ^(outer)) and the second coordinate (u_(ref _N) ^(inner),v_(ref _N) ^(inner)) in the reference image 710, as follows:

$\left( {u_{ref\_ N},v_{ref\_ N}} \right) = \left( {\frac{u_{{ref}\_ N}^{outer} + u_{ref\_ N}^{inner}}{2},\frac{v_{{ref}\_ N}^{outer} + v_{{ref}\_ N}^{inner}}{2}} \right)$

In an embodiment, the reference image coordinate for the verificationsymbol may be its center coordinate, and determining the centercoordinate of the verification symbol 530 based on the respective centercoordinates of the first shape 735 and the second shape 735 may improvea robustness of the verification process against image noise. Forinstance, image noise may introduce error in the determination of acenter coordinate for the first shape 735, but not to the determinationof the center coordinate for the second shape 737. In some cases, thesecond shape 737 may have may in reality share the same center locationas the first shape 735, but image noise may cause the center coordinateof the second shape 737 to appear in an image to be different than thecenter coordinate of the first shape 735. In this scenario, simply usingthe center coordinate of the second shape 737 as the center coordinateof the verification symbol 530 may lead to an undesirable amount oferror. The amount of error may be reduced by using an average of thecenter coordinate for the first shape 735 and the center coordinate forthe second shape 737 as the center coordinate of the verification symbol530.

In an embodiment, the one or more reference locations discussed abovemay be a plurality of reference locations that respectively correspondwith a plurality of reference image coordinates. In this embodiment, thereference image coordinate may be one of the plurality of referenceimage coordinates. For instance, as illustrated in FIGS. 6A and 6B,there may be multiple reference locations such as the referencelocations 610A-610I to which the verification symbol 530 is moved orotherwise placed. For each of the reference locations 610A-610I of theverification symbol 530, the robot control system 110 may retrieve orotherwise receive a respective reference image captured by the camera570 of the verification symbol 530 at that location, and may determine arespective reference image coordinate that indicates where theverification symbol 530 appears in the respective reference image.

Returning to FIG. 4A, the method 400 may further include step 409, inwhich the control circuit 111 controls, based on the camera calibrationinformation, movement of the robot arm to perform a robot operation. Inan embodiment, this step may involve the control circuit 111 generatinga second movement command that is based on the camera calibrationinformation, and outputting the second movement command to thecommunication interface 113. The communication interface 113 may in turncommunicate the second movement command to the robot to control movementof the robot arm. For instance, as illustrated in FIG. 5A, after thefirst camera calibration, the robot control system 110 controls therobot 550 to perform a robot operation involving robot tasks, such aspicking up objects 582A, 582B, and 582C. The movement of the robot 550may be based on the camera calibration information obtained from thefirst camera calibration and based on images of the objects 582A, 582B,582C captured by the camera 570.

In step 411, the control circuit 111 detects an idle period during therobot operation. In an aspect, the idle period of robot may be a timeperiod during which the robot is free from performing a robot taskduring the robot operation. In some cases, if the robot operation isbased on picking up objects from the conveyor belt 573, the idle periodmay be based on an absence of objects on the conveyor belt 573. Morespecifically, the conveyor belt 573 may be reachable by the robot arm,and the control circuit 111 is configured to detect the idle period bydetecting that there are no objects on the conveyor belt 573, or that adistance between the robot 550 and a closest object on the conveyor belt573 exceeds a defined distance threshold. In some cases, the controlcircuit 111 may receive a signal indicating that an idle period is aboutto occur, where the signal may be received from another device orcomponent monitoring the robot operation. For instance, as illustratedin FIG. 5A, the robot control system 110 may detect an idle periodbetween a robot task involving picking up the second object 582B and arobot task involving picking up the third object 582C during the robotoperation because a large distance exists between the second object 582Band the third object 582C. During this idle period, after the robot 550picks up the second object 582B, it may have an idle period during whichit is free from performing a robot task, because the object 582C is notyet reachable by the robot 550. In one example, the robot control system110 may detect the idle period when no object on the conveyer belt 573is reachable by the robot 550 and/or when the robot control system 110determines that a distance between the robot 550 and the closest object(e.g., third object 582C) upstream on the conveyer belt 573 exceeds acertain threshold.

Returning to FIGS. 4A and 4B, the method 400 may further include a step451, in which the control circuit 111 controls the robot arm to move theverification symbol 530, during the idle period, to at least thereference location that was used in step 403 (which was used to capturethe reference image). In an embodiment, step 451 may involve the controlcircuit 111 generating a third movement command, and outputting thethird movement command to the communication interface 113. Thecommunication interface 113 may be configured to then communicate thethird movement to the robot to cause the robot arm to move based on themovement command. In some cases, the third movement command may involvea set of stored motor commands that correspond to the referencelocation. In some cases, the third movement command may be generatedbased on the camera calibration information from step 401. In othercases, the third movement command in step 451 does not rely on thecamera calibration information from step 401.

In step 453, the control circuit 111 retrieves or otherwise receives anadditional image of the verification symbol (e.g., 530) from the camera(e.g., 570) during the idle period, wherein the additional image is averification image for the verification, and is an image of theverification symbol at least at the reference location during the idleperiod. That is, the verification image for the reference location iscaptured while the verification symbol (e.g., 530) is or was at thereference location. In an embodiment, step 453 involves the controlcircuit 111 generating a camera command that controls the camera (e.g.,570) to capture the verification image. The control circuit 111 mayoutput the camera command to the communication interface 113, which maycommunicate the camera command to the camera (e.g., 570). In anembodiment, step 451 may involve controlling the robot arm to move theverification symbol to multiple reference locations, and receivingmultiple respective verification images captured by the camera. Forinstance, as illustrated in FIGS. 6A and 6B, during an idle period, therobot control system 110 may control the robot arm of the robot 550 tomove the verification symbol 530 to one of the reference locations610A-610I and capture an image of the verification symbol 530 at thelocation as a verification image. If the idle period is not yet over, ormore specifically if a sufficient amount of time remains in the idleperiod, the robot control system 110 may control the robot arm of therobot 550 to move the verification symbol 530 to another one of thereference locations 610A-610I and capture an image of the verificationsymbol 530 at that other location as another verification image. If theidle period ends, the robot control system 110 may stop capturingverification images. As such, during each idle period, the robot controlsystem 110 may control the robot arm of the robot 550 to move theverification symbol 530 to one or more of the reference locations610A-610I and capture a verification image at each of the one or more ofthe reference locations 610A-610I.

Returning to FIG. 4B, the method 400 may further include step 455, inwhich the control circuit 111 determines a verification image coordinateused for the verification, the verification image coordinate being acoordinate at which the verification symbol appears in the verificationimage. If the verification symbol (e.g., 530) is moved to a plurality ofreference locations (e.g., 610A-610I), the camera (e.g., 570) maycapture a plurality of verification images that respectively correspondwith the plurality of reference locations, and the control circuit 111may determine a plurality of verification image coordinates thatrespectively correspond with the plurality of verification images andrespectively correspond with the plurality of reference locations. Theplurality of verification images may be all captured by the camera(e.g., 570) in a single idle period (e.g., if the single idle period issufficiently long to allow the robot arm to move the verification symbol(e.g., 530) to all of the reference locations 610A-610I), or in severaldifferent idle periods (e.g., if each of the idle periods is not longenough for the robot arm to move the verification symbol 530 to all ofthe reference locations 610A-610I).

In an embodiment, the verification image coordinate may be determined ina manner similar to that for the reference image coordinate. Forinstance, the verification image coordinate may be a center coordinateof the verification symbol (e.g., 530), and may be determined as anaverage of a center coordinate of a first shape of the verificationsymbol (e.g., 530) and a center coordinate of a second shape of theverification symbol in the verification image (e.g., 760). For instance,FIG. 7B shows a verification image 760 captured at the referencelocation N of the reference locations. The verification image 760 showsa verification portion 780, which is an image portion in theverification image 760 showing the verification symbol 530. The robotcontrol system 110 may identify, from the verification portion 780, afirst shape 785 that is the same as or substantially the same as thefirst shape 585 of the verification symbol 530 of FIG. 5C. The robotcontrol system 110 may further identify, from the verification portion780, a second shape 787 that is the same or substantially the same asthe second shape 587 of the verification symbol 530. Further, the robotcontrol system 110 may be configured to determine a center coordinate(u_(verify _N) ^(outer), v_(verify _N) ^(outer)) of the first shape 785shown in the verification portion 780 of the verification image 760, anddetermine a center coordinate (u_(verify _N) ^(inner), v_(verify _N)^(inner)) of the second shape 787 shown in the verification portion 780of the verification image 760. The robot control system 110 may furtherdetermine the verification image coordinate (u_(verify _N),v_(verify _N)) for the verification image 760 as an average of thecenter coordinate of the first shape 785 and the center coordinate ofthe second shape 787 in the verification image 760, as follows:

$\left( {u_{verify\_ N},v_{{ve}{rify\_ N}}} \right) = \left( {\frac{u_{{verify}\_ N}^{outer} + u_{verify\_ N}^{inner}}{2},\frac{v_{{verify}\_ N}^{outer} + v_{{verify}\_ N}^{inner}}{2}} \right)$

Returning to FIG. 4B, the method 400 may further include step 457, inwhich the control circuit 111 determines a deviation parameter valuebased on an amount of deviation between the reference image coordinateof step 403 and the verification image coordinate of step 455, whereinthe reference image coordinate and the verification image coordinate areboth associated with the reference location N. In one example, thedeviation between the reference image coordinate and the verificationimage coordinate may be a distance between the reference imagecoordinate and the verification image coordinate. For instance, giventhat the reference image coordinate at the reference location N isexpressed as (u_(ref _N), v_(ref _N)) and the verification imagecoordinate at the reference location N is expressed as (u_(verify _N),v_(verify _N), a deviation (e.g., distance) at the reference location Nmay be expressed as (u_(ref _N)−u_(verify _N)²+(v_(ref _N)−v_(verify _N))².

As discussed above, in an aspect where the one or more referencelocations are a plurality of reference locations, the control circuit111 may be configured to determine the plurality of verification imagecoordinates respectively corresponding to the plurality of referencelocations, where the verification image coordinate discussed above isone of the plurality of verification image coordinates. In such anaspect, the deviation parameter value is based on respective amounts ofdeviation between the plurality of reference image coordinates and theplurality of verification image coordinates for the plurality ofreference locations, wherein each amount of deviation of the respectiveamounts of deviation is between: (a) a reference image coordinatecorresponding to a respective reference location of the plurality ofreference locations, and (b) a verification image coordinatecorresponding to the same reference location. The plurality ofverification image coordinates may be respective coordinates at whichthe verification symbol appears in a plurality of verification images,the verification image discussed above being one of the plurality ofverification images. The control circuit 111 may be configured tocontrol the camera to capture all of the plurality of verificationimages in one idle period, and/or may be configured to control thecamera to capture the plurality of verification images in different idleperiods.

For instance, when there are multiple reference locations, as shown inFIGS. 6A and 6B, the robot control system 110 may determine a pluralityof respective reference image coordinates corresponding to the pluralityof reference locations, and determine a plurality of respectiveverification image coordinates corresponding to the plurality ofreference locations, and determine respective amounts of deviationbetween the plurality of reference image coordinates and the pluralityof verification image coordinates. The deviation parameter value may bebased on the respective amounts of deviation between the plurality ofreference image coordinates and the plurality of verification imagecoordinates. For instance, the deviation parameter may be an average ofthe respective amounts of deviation, as follows.

${{deviation}\mspace{14mu}{parameter}} = {\frac{\sum\limits_{N = 1}^{M}\sqrt{\left( {u_{ref\_ N} - u_{verify\_ N}} \right)^{2} + \left( {v_{ref\_ N} - v_{verify\_ N}} \right)^{2}}}{M}.}$

In the above expression, N may refer to the Nth reference location,while M may refer to a total number of reference locations.

Returning to FIG. 4B, the method 400 may further include step 459, inwhich the control circuit 111 determines whether the deviation parametervalue exceeds a defined threshold (which may also be referred to as adefined deviation threshold). Further, in step 461, the control circuit111 may, in response to a determination that the deviation parametervalue exceeds the defined threshold, perform a second camera calibrationto determine updated camera calibration information for the camera. Forinstance, the deviation parameter value exceeding the defined thresholdmay indicate that the camera calibration information of the camera isoutdated and/or is likely to cause an undesirable amount of errors in arobot operation. Hence, if the deviation parameter value exceeds thedefined threshold, a second camera calibration for the camera may beperformed to update the camera calibration information for the camera(e.g., 570). The second camera calibration may use the same techniquesas the first camera calibration, but may be based on images that aremore recently captured by the camera. In an example, the robot operationmay be stopped or paused if step 459 indicates that the deviationparameter value exceeds the defined threshold, and then proceed toperform the second camera calibration, which may begin by capturingimages for the second camera calibration. After the second cameracalibration is complete and the camera calibration information for thecamera is updated, the robot control system 110 may resume the robotoperation using the updated camera calibration information.

In an embodiment, the control circuit 111 may be configured, in responseto a determination that the deviation parameter value does not exceedthe defined threshold, to control the robot to continue the robotoperation after the idle period without performing additional cameracalibration (e.g., by outputting a fourth movement command to the robotvia the communication interface). Such a condition may indicate that thecamera calibration information from step 401 is still sufficientlyaccurate, and that robot operation can continue without experiencing anundesirable amount of errors.

In an embodiment, the control circuit 111 may be configured to determinea temperature of an environment in which the robot is located, and toadjust at least one of the defined deviation threshold (also referred toas re-defining the deviation threshold) or the camera calibrationinformation for the camera based on the measured temperature. Forexample, the control circuit 111 may determine the temperature of theenvironment by measuring the temperature or receiving temperature datafrom another device or component. In such an embodiment, the controlcircuit 111 may be configured to adjust the defined threshold based onthe measured temperature by: setting the defined threshold to have afirst value when the measured temperature is outside of a defined range,and setting the threshold to a have a second value lower than the firstvalue when the measured temperature is within the defined range. Forinstance, an excessively high temperature or an excessively lowtemperature may cause changes in the camera. More specifically, atemperature change may affect the intrinsic parameters of the camera.For example, components in the camera may expand when the temperatureincreases and may contract when the temperature decreases, which mayaffect the intrinsic parameters of the camera. Therefore, it may beadvantageous to adjust the defined deviation threshold based on thetemperature or amount of temperature change. For example, when thetemperature is within a range of a normal operating temperature (e.g., adefined range based around room temperature), then the defined deviationthreshold may be lower, because the temperature does not adverselyaffect the camera. On the other hand, when the temperature is outsidethe range of a normal operating temperature, the deviation threshold maybe higher because a cold or hot temperature adversely affects thecamera. In an alternative example, the deviation threshold may bedefined to a lower value when the temperature is outside of a normaloperating temperature, so as to more frequently trigger additionalcamera calibration. In this example, the deviation threshold may bedefined to a higher value when the temperature is within the normaloperating temperature, so as to less frequently trigger additionalcamera calibration.

FIG. 8 depicts an example time line 800 where the camera calibration andthe verification of the camera calibration are performed. Before a robotoperation begins, the robot control system 110 of FIG. 1A or 1B performsthe first camera calibration to determine camera calibration informationfor the camera (e.g., camera 570 of FIG. 5A or 5B) during a calibrationperiod 811. After the first camera calibration is complete, the robotcontrol system 110 captures reference images of the verification symbol(e.g., verification symbol 530) at various reference locations anddetermines reference image coordinates at which the verification symbolappears in respective reference images (e.g., reference image 710 ofFIG. 7A), during a reference acquisition period 813. The robot operationmay begin after the reference acquisition period 813 ends upondetermining the reference image coordinates.

After the robot operation begins, during a task period 815, the robotcontrol system 110 controls the robot (e.g., robot 550 of FIG. 5A or 5B)to perform one or more robot tasks and thus, in an embodiment, may notbe able to collect verification images (e.g., verification image 760 ofFIG. 7B). The robot control system 110 detects an idle period 817 duringwhich the robot is free from performing a robot task after the taskperiod 815. Hence, during the idle period 817, the robot control system110 captures one or more verification images of the verification symbolat a first set of one or more locations (e.g., 610A-610B) of thereference locations, respectively. After the idle period 817 ends,during a task period 819, the robot control system 110 resumescontrolling the robot to perform one or more robot tasks and thus maynot collect verification images. The robot control system 110 detects anidle period 821 during which the robot is free from performing a robottask, after the task period 817. During the idle period 821, the robotcontrol system 110 captures one or more verification images of theverification symbol at a second set of one or more locations (e.g.,610C-610E) of the reference locations, respectively. After the idleperiod 821, during a robot task period 823, the robot control system 110resumes controlling the robot to perform one or more robot tasks andthus may not collect verification images. The robot control system 110detects an idle period 825 during which the robot is free fromperforming a robot task after the task period 823. During the idleperiod 825, the robot control system 110 captures one or moreverification images of the verification symbol at a third set of one ormore locations (e.g., 610E-610I) of the reference locations,respectively.

The verification images (e.g., 760) captured during the idle periods817, 821, and 825 may be captured at different respective locations ofthe reference locations. For instance, the first set, second set, andthird set of one or more locations may be different from each other, andmay have no overlap in locations. Further, during the idle period 825,the robot control system 110 may determine that the verification imagecapture is complete, which may indicate that a sufficient number ofverification images are captured for verification of the cameracalibration. In one embodiment, the robot control system 110 maydetermine that the verification image capture is complete ifverification images are captured at all of the reference locations(e.g., 610A-610I). In one embodiment, the robot control system 110 maydetermine that the verification image capture is complete if a number ofthe verification images reaches a defined target count.

Upon determining that the verification image capture is complete, therobot control system 110 determines the verification image coordinatesat which the verification symbol appears in the respective verificationimages. Subsequently, the robot control system 110 determines adeviation parameter value based on respective amounts of deviation ofthe verification image coordinates from the reference image coordinates.If the deviation parameter exceeds a defined threshold, the robotcontrol system 110 performs another camera calibration. In this example,however, the deviation parameter does not exceed the defined threshold,and thus the robot control system 110 continues performing a robot taskduring a task duration 827 after the idle period 825, without performingadditional camera calibration.

FIG. 9 depicts an example flow diagram 900 that shows a verificationprocess related to the timeline in FIG. 8. At step 901, the robotcontrol system 110 of FIG. 1A, 1B, or 1C performs the first cameracalibration of the camera (e.g., camera 570 of FIG. 5A or 5B) todetermine camera calibration information of the camera. At step 903, therobot control system 110 controls the robot (e.g., robot 550 of FIG. 5Aor 5B) to move the verification symbol (e.g., verification symbol 530 ofFIG. 5A or 5B) to reference locations and captures, via the camera,respective instances of the reference images (e.g., reference image 710of FIG. 7A) of the verification symbol at respective referencelocations. At step 905, the robot control system 110 begins a robotoperation of the robot based on the camera calibration informationobtained from the first camera calibration.

At step 907, the robot control system 110 detects an idle period duringthe robot operation. At step 909, the robot control system 110 controlsthe robot (e.g., robot 550 of FIG. 5A or 5B) to move the verificationsymbol (e.g., verification symbol 530 of FIG. 5A or 5B) to one or morelocations of the reference locations during the idle period andcaptures, via the camera, one or more verification images (e.g.,verification image 760 of FIG. 7B) respectively at the one or morelocations of the reference locations. In some cases, the robot controlsystem 110 may control the robot to move the verification symbol to asmany reference locations as is permitted by a duration of the idleperiod. At step 911, the robot control system 110 determines whether atotal number of the captured verification images has reached a definedtarget count. If the total number of the captured verification imageshas not reached the target count, the robot control system 110 attemptsto detect another, subsequent idle period during the robot operation, byreturning to step 907, to capture more verification images.

If the total number of the captured verification images has reached thetarget count, at step 913, the robot control system 110 performsverification of the camera calibration based on the reference images(e.g., 710) and the verification images (e.g. 760). The verification ofthe camera calibration produces a deviation parameter. At step 915, therobot control system 110 determines whether the deviation parameterexceeds a defined threshold. If the deviation parameter does not exceedthe threshold, the robot control system 110 may reset the total numberof captured verification images to zero at step 919 and may continue therobot operation after the idle period while attempting to detect anotheridle period to capture a new set of verification images, by returning tostep 907.

If the deviation parameter exceeds the threshold, the robot controlsystem 110 may stop the robot operation and perform a second cameracalibration, at step 917. After the second camera calibration at 917,the robot control system 110 may reset the total number of capturedverification images to zero at 921. After step 921, the flow diagram mayreturn to step 903, where the robot control system 110 controls therobot (e.g., 550) to move the verification symbol (e.g., 530) to thereference locations and captures, via the camera (e.g., 570), a new setof reference images (e.g., 710) of the verification symbol at therespective reference locations, such that the new set of referenceimages may be used for verification later.

Additional Discussion of Various Embodiments

Embodiment 1 relates to a robot control system comprising acommunication interface configured to communicate with a robot having abase and a robot arm with a verification symbol disposed thereon, and tocommunicate with a camera having a camera field of view. The robotcontrol system further comprises a control circuit configured to performa first camera calibration to determine camera calibration informationassociated with the camera. The control circuit is further configured:a) to control the robot arm to move the verification symbol, during orafter the first camera calibration, to a location within the camerafield of view by outputting a first movement command to the robot viathe communication interface, the location being a reference location ofone or more reference locations for verification of the first cameracalibration, b) to receive an image of the verification symbol from thecamera via the communication interface, wherein the camera is configuredto capture the image of the verification symbol at the referencelocation, the image being a reference image for the verification, c) todetermine a reference image coordinate for the verification, thereference image coordinate being a coordinate at which the verificationsymbol appears in the reference image; d) to control, based on thecamera calibration information, movement of the robot arm to perform arobot operation by outputting a second movement command that is based onthe camera calibration information to the robot via the communicationinterface; e) to detect an idle period during the robot operation; f) tocontrol the robot arm to move the verification symbol to at least thereference location during the idle period by outputting a third movementcommand to the robot via the communication interface; g) to receive anadditional image of the verification symbol from the camera via thecommunication interface during the idle period, wherein the camera isconfigured to capture the additional image of the verification symbol atleast at the reference location, the additional image being averification image for the verification; h) to determine a verificationimage coordinate used for the verification, the verification imagecoordinate being a coordinate at which the verification symbol appearsin the verification image; i) to determine a deviation parameter valuebased on an amount of deviation between the reference image coordinateand the verification image coordinate, the reference image coordinateand the verification image coordinate both associated with the referencelocation, wherein the deviation parameter value is indicative of achange in the camera since the first camera calibration or of a changein a relationship between the camera and the robot since the firstcamera calibration, i) to determine whether the deviation parametervalue exceeds a defined threshold, and j) in response to a determinationthat the deviation parameter value exceeds the defined threshold, toperform a second camera calibration to determine updated cameracalibration information.

Embodiment 2 includes the robot control system of embodiment 1, whereinthe control circuit is configured, in response to a determination thatthe deviation parameter value does not exceed the defined threshold, tocontrol the robot to continue the robot operation after the idle periodwithout performing additional camera calibration by outputting a fourthmovement command to the robot via the communication interface.

Embodiment 3 includes the robot control system of embodiment 1 or 2,wherein the one or more reference locations are a plurality of referencelocations that respectively correspond with a plurality of referenceimage coordinates, the reference image coordinate being one of theplurality of reference image coordinates. In this embodiment, thecontrol circuit is further configured to determine a plurality ofverification image coordinates respectively corresponding to theplurality of reference locations, wherein the verification imagecoordinate is one of the plurality of verification image coordinates,and wherein the deviation parameter value is based on respective amountsof deviation between the plurality of reference image coordinates andthe plurality of verification image coordinates for the plurality ofreference locations, wherein each amount of deviation of the respectiveamounts of deviation is between: (a) a reference image coordinatecorresponding to a respective reference location of the plurality ofreference locations, and (b) a verification image coordinatecorresponding to the same reference location.

Embodiment 4 includes the robot control system of embodiment 3, whereinthe plurality of verification image coordinates are respectivecoordinates at which the verification symbol appears in a plurality ofverification images, the verification image being one of the pluralityof verification images, and wherein the control circuit is configured tocontrol the camera to capture all of the plurality of verificationimages in the idle period.

Embodiment 5 includes the robot control system of embodiment 3, whereinthe plurality of verification image coordinates are respectivecoordinates at which the verification symbol appears in a plurality ofverification images, the verification image being one of the pluralityof verification images, and wherein the control circuit is configured tocontrol the camera to capture the plurality of verification images indifferent idle periods, the idle period being one of the different idleperiods.

Embodiment 6 includes the robot control system of any one of embodiments1-5, wherein the verification symbol includes a first region having afirst color and a second region having a second color, wherein a ratioof an area of the first region to an area of the second region isdefined and stored on a storage device of the robot control system as adefined ratio.

Embodiment 7 includes the robot control system of embodiment 6, whereinthe control circuit is configured to identify the verification symbol inthe reference image or the verification image based on the definedratio.

Embodiment 8 includes the robot control system of embodiment 7, whereinthe robot arm has a calibration pattern disposed thereon, wherein thereference image includes the verification symbol and the calibrationpattern, wherein the control circuit is configured to determine whethera portion of the reference image is the verification symbol or thecalibration pattern by determining whether the portion of the referenceimage has a first image region having the first color and has a secondimage region having the second color, and whether a ratio between anarea of the first image region and an area of the second image region isequal to the defined ratio.

Embodiment 9 includes the robot control system of any one of embodiments1-8, wherein the verification symbol includes a first shape and a secondshape that are concentric with each other, wherein respective centers ofthe first shape and the second shape are at substantially the samelocation.

Embodiment 10 includes the robot control system of embodiment 9, whereinthe control circuit is configured to determine the reference imagecoordinate by: a) determining a first coordinate of a center of thefirst shape in the reference image; b) determining a second coordinateof a center of the second shape in the reference image; and c)determining the reference image coordinate as an average of the firstcoordinate and the second coordinate in the reference image. In thisembodiment, the control circuit is configured to determine theverification image coordinate by: d) determining a first coordinate of acenter of the first shape in the verification image; e) determining asecond coordinate of a center of the second shape in the verificationimage; and f) determining the verification image coordinate as anaverage of the first coordinate and the second coordinate in theverification image.

Embodiment 11 includes the robot control system of any one ofembodiments 1-10, wherein the control circuit is configured to identifythe verification symbol in the reference image or the verification imageby identifying a circular ring, the verification symbol being shaped asthe circular ring.

Embodiment 12 includes the robot control system of any one ofembodiments 1-11, wherein the control circuit is further configured todetermine a temperature of an environment in which the robot is located;and to adjust at least one of the defined threshold or the cameracalibration information based on the temperature that is measured.

Embodiment 13 includes the robot control system of embodiment 12,wherein the control circuit is configured to adjust the definedthreshold based on the temperature by: setting the defined threshold tohave a first value when the temperature is outside of a defined range;and setting the threshold to a have a second value lower than the firstvalue when the temperature is within the defined range.

Embodiment 14 includes the robot control system of any one ofembodiments 1-13, wherein the one or more reference locations to whichthe control circuit is configured to cause the verification symbol to bemoved via the robot arm include a plurality of reference locationsdisposed on a surface of a sphere that is concave with respect to thecamera.

Embodiment 15 includes the robot control system of embodiment 14,wherein the control circuit is further configured to control the robotarm to move the verification symbol to be tangent to the surface of thesphere at each reference location of the plurality of referencelocations.

Embodiment 16 includes the robot control system of any one ofembodiments 1-15, wherein the control circuit is configured to controlthe robot arm to move the verification symbol to directly face thecamera when the verification symbol is moved to the reference location.

Embodiment 17 includes the robot control system of any one ofembodiments 1-16, wherein the control circuit is configured to detectthe idle period of the robot operation by detecting a time period duringwhich the robot is free from performing a robot task during the robotoperation.

Embodiment 18 includes the robot control system of embodiment 17,wherein the control circuit is configured to control the robot arm tointeract with objects on a conveyor belt that is reachable by the robotarm, wherein the control circuit is configured to detect the idle periodby detecting the conveyor belt having no object thereon, or detectingthat a distance between the robot and a closest object on the conveyorbelt exceeds a defined distance threshold.

While various embodiments have been described above, it should beunderstood that they have been presented only as illustrations andexamples of the present invention, and not by way of limitation. It willbe apparent to persons skilled in the relevant art that various changesin form and detail can be made therein without departing from the spiritand scope of the invention. Thus, the breadth and scope of the presentinvention should not be limited by any of the above-described exemplaryembodiments, but should be defined only in accordance with the appendedclaims and their equivalents. It will also be understood that eachfeature of each embodiment discussed herein, and of each reference citedherein, can be used in combination with the features of any otherembodiment. All patents and publications discussed herein areincorporated by reference herein in their entirety.

What is claimed is:
 1. A robot control system comprising: acommunication interface configured to communicate with a robot having averification symbol disposed thereon, and to communicate with a camerahaving a camera field of view; and a control circuit configured toperform a first camera calibration to determine camera calibrationinformation associated with the camera, to receive an image representingthe verification symbol from the camera via the communication interface,wherein the image representing the verification symbol is captured whenthe verification symbol is at a reference location of one or morereference locations used for verification of the first cameracalibration, the image being a reference image for the verification ofthe first camera calibration, to determine a reference image coordinatefor the verification symbol, the reference image coordinate being acoordinate at which the verification symbol appears in the referenceimage; to output a movement command for controlling movement of therobot to perform a robot operation, wherein the movement command isbased on the camera calibration information; to output an additionalmovement command for controlling the robot to move the verificationsymbol to at least the reference location; to receive an additionalimage representing the verification symbol from the camera via thecommunication interface, wherein the additional image is captured whenthe verification symbol is at the reference location, the additionalimage being a verification image for the verification of the firstcamera calibration; to determine a verification image coordinate usedfor the verification, the verification image coordinate being acoordinate at which the verification symbol appears in the verificationimage; to determine a deviation parameter value based on an amount ofdeviation between the reference image coordinate and the verificationimage coordinate, the reference image coordinate and the verificationimage coordinate both being associated with the reference location, todetermine, based on the deviation parameter value, whether to update thecamera calibration information, and in response to a determination toupdate the camera calibration information, to perform a second cameracalibration to determine updated camera calibration information.
 2. Therobot control system of claim 1, wherein the additional movement commandis a first additional movement command, wherein the control circuit isconfigured, in response to a determination not to update the cameracalibration information, to output a second additional movement commandfor controlling the robot to continue the robot operation withoutperforming additional camera calibration.
 3. The robot control system ofclaim 1, where the control circuit is further configured to detect anidle period during the robot operation, and wherein the additionalmovement command for controlling the robot to move the verificationsymbol to at least the reference location is output during the idleperiod.
 4. The robot control system of claim 3, wherein the controlcircuit is configured to detect the idle period of the robot operationby detecting a time period during which the robot is free fromperforming a robot task during the robot operation.
 5. The robot controlsystem of claim 3, wherein the one or more reference locations are aplurality of reference locations that respectively correspond with aplurality of reference image coordinates, the reference image coordinatebeing one of the plurality of reference image coordinates, wherein thecontrol circuit is further configured to determine a plurality ofverification image coordinates respectively corresponding to theplurality of reference locations, wherein the verification imagecoordinate is one of the plurality of verification image coordinates,and wherein the deviation parameter value is based on respective amountsof deviation between the plurality of reference image coordinates andthe plurality of verification image coordinates for the plurality ofreference locations, wherein each amount of deviation of the respectiveamounts of deviation is between: (a) a reference image coordinatecorresponding to a respective reference location of the plurality ofreference locations, and (b) a verification image coordinatecorresponding to the same reference location.
 6. The robot controlsystem of claim 5, wherein the plurality of verification imagecoordinates are respective coordinates at which the verification symbolappears in a plurality of verification images, the verification imagebeing one of the plurality of verification images, and wherein thecontrol circuit is configured to control the camera to capture all ofthe plurality of verification images in the idle period.
 7. The robotcontrol system of claim 5, wherein the plurality of verification imagecoordinates are respective coordinates at which the verification symbolappears in a plurality of verification images, the verification imagebeing one of the plurality of verification images, and wherein thecontrol circuit is configured to control the camera to capture theplurality of verification images in different idle periods, the idleperiod being one of the different idle periods.
 8. The robot controlsystem of claim 1, wherein the verification symbol includes a firstregion having a first color and a second region having a second color,wherein a ratio of an area of the first region to an area of the secondregion is defined and stored on a storage device of the robot controlsystem as a defined ratio.
 9. The robot control system of claim 8,wherein the control circuit is configured to identify the verificationsymbol in the reference image or the verification image based on thedefined ratio.
 10. The robot control system of claim 9, wherein therobot has a calibration pattern disposed thereon, wherein the referenceimage represents the verification symbol and the calibration pattern,wherein the control circuit is configured to determine whether a portionof the reference image is the verification symbol or the calibrationpattern by determining whether the portion of the reference image has afirst image region having the first color and has a second image regionhaving the second color, and whether a ratio between an area of thefirst image region and an area of the second image region is equal tothe defined ratio.
 11. The robot control system of claim 1, wherein theverification symbol includes a first shape and a second shape that areconcentric with each other, wherein respective centers of the firstshape and the second shape are at substantially the same location. 12.The robot control system of claim 11, wherein the control circuit isconfigured to determine the reference image coordinate by: determining afirst coordinate of a center of the first shape in the reference image;determining a second coordinate of a center of the second shape in thereference image; and determining the reference image coordinate as anaverage of the first coordinate and the second coordinate in thereference image, and wherein the control circuit is configured todetermine the verification image coordinate by: determining a firstcoordinate of a center of the first shape in the verification image;determining a second coordinate of a center of the second shape in theverification image; and determining the verification image coordinate asan average of the first coordinate and the second coordinate in theverification image.
 13. The robot control system of claim 1, wherein thecontrol circuit is configured to identify the verification symbol in thereference image or the verification image by identifying a circularring, the verification symbol being shaped as the circular ring.
 14. Therobot control system of claim 1, wherein the control circuit isconfigured to determine to update the camera calibration informationwhen the deviation parameter value exceeds a defined threshold, andwherein the deviation parameter value is indicative of a change in thecamera since the first camera calibration or of a change in arelationship between the camera and the robot since the first cameracalibration.
 15. The robot control system of claim 14, wherein thecontrol circuit is further configured to determine a temperature of anenvironment in which the robot is located; and adjust at least one ofthe defined threshold or the camera calibration information based on thetemperature that is determined.
 16. The robot control system of claim15, wherein the control circuit is configured to adjust the definedthreshold based on the temperature by: setting the defined threshold tohave a first value when the temperature is outside of a defined range;and setting the threshold to a have a second value lower than the firstvalue when the temperature is within the defined range.
 17. The robotcontrol system of claim 1, wherein the one or more reference locationsto which the control circuit is configured to cause the verificationsymbol to be moved via the robot include a plurality of referencelocations disposed on a surface of a sphere that is concave with respectto the camera.
 18. The robot control system of claim 17, wherein thecontrol circuit is further configured to control the robot to move theverification symbol to be tangent to the surface of the sphere at eachreference location of the plurality of reference locations.
 19. A methodof performing camera calibration verification for robot control, themethod comprising: performing, by a robot control system, a first cameracalibration to determine camera calibration information, wherein therobot control system comprises a communication interface configured tocommunicate with a robot having a verification symbol disposed thereon,and to communicate with a camera having a camera field of view andassociated with the camera calibration information; receiving, by therobot control system and from the communication interface, an imagerepresenting the verification symbol, wherein the communicationinterface is configured to receive the image from the camera, whereinthe image representing the verification symbol is captured when theverification symbol is at a reference location of one or more referencelocations used for verification of the first camera calibration, theimage being a reference image for the verification of the first cameracalibration; determining, by the robot control system, a reference imagecoordinate for the verification symbol, the reference image coordinatebeing a coordinate at which the verification symbol appears in thereference image; outputting, by the robot control system, a movementcommand for controlling the robot to perform a robot operation, whereinthe movement command is based on the camera calibration information;outputting, by the robot control system, an additional movement commandfor controlling the robot to move the verification symbol to at leastthe reference location; receiving, by the robot control system and fromthe communication interface, an additional image representing theverification symbol from the camera via the communication interface,wherein the additional image is captured when the verification symbol isat the reference location, the additional image being a verificationimage for the verification of the first camera calibration; determining,by the robot control system, a verification image coordinate used forthe verification, the verification image coordinate being a coordinateat which the verification symbol appears in the verification image;determining, by the robot control system, a deviation parameter valuebased on an amount of deviation between the reference image coordinateand the verification image coordinate, the reference image coordinateand the verification image coordinate both being associated with thereference location; determining, by the robot control system, based onthe deviation parameter value, whether to update the camera calibrationinformation; and performing by the robot control system, in response toa determination to update the camera calibration information, a secondcamera calibration to determine updated camera calibration information.20. A non-transitory computer-readable medium having instructions storedthereon that, when executed by a control circuit of a robot controlsystem, causes the control circuit to perform a first camera calibrationto determine camera calibration information, wherein the robot controlsystem comprises a communication interface configured to communicatewith a robot having a verification symbol disposed thereon, and tocommunicate with a camera having a camera field of view and associatedwith the camera calibration information; to receive, from thecommunication interface, an image representing the verification symbol,wherein the communication interface is configured to receive the imagefrom the camera, wherein the image representing the verification symbolis captured when the verification symbol is at a reference location ofone or more reference locations used for verification of the firstcamera calibration, the image being a reference image for theverification; to determine a reference image coordinate for theverification symbol, the reference image coordinate being a coordinateat which the verification symbol appears in the reference image; tooutput a movement command for controlling the robot to perform a robotoperation, wherein the movement command is based on the cameracalibration information; to output an additional movement command forcontrolling the robot to move the verification symbol to at least thereference location; to receive, from the communication interface, anadditional image representing the verification symbol from the cameravia the communication interface, wherein the additional image iscaptured when the verification symbol is at the reference location, theadditional image being a verification image for the verification of thefirst camera calibration; to determine a verification image coordinateused for the verification, the verification image coordinate being acoordinate at which the verification symbol appears in the verificationimage; to determine a deviation parameter value based on an amount ofdeviation between the reference image coordinate and the verificationimage coordinate, the reference image coordinate and the verificationimage coordinate both being associated with the reference location; todetermine, based on the deviation parameter value, whether to update thecamera calibration information; and to perform, in response to adetermination to update the camera calibration information, a secondcamera calibration to determine updated camera calibration information.